Robotic CCD microscope for enhanced crystal recognition

ABSTRACT

A robotic CCD microscope and procedures to automate crystal recognition. The robotic CCD microscope and procedures enables more accurate crystal recognition, leading to fewer false negative and fewer false positives, and enable detection of smaller crystals compared to other methods available today.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 60/662,702 filed Mar. 16, 2005 and titled “Robotic CCDMicroscope for Enhanced Crystal Recognition.” U.S. Provisional PatentApplication No. 60/662,702 filed Mar. 16, 2005 and titled “Robotic CCDMicroscope for Enhanced Crystal Recognition” is incorporated herein bythis reference.

The United States Government has rights in this invention pursuant toContract No. W-7405-ENG-48 between the United States Department ofEnergy and the University of California for the operation of LawrenceLivermore National Laboratory.

BACKGROUND

1. Field of Endeavor

The present invention relates to charge-coupled devices (CCD) and moreparticularly to a robotic CCD microscope for enhanced crystalrecognition.

2. State of Technology

U.S. Pat. No. 5,597,457 for a system and method for forming syntheticprotein crystals to determine the conformational structure bycrystallography to George D. Craig, issued Jan. 28, 1997 provides thefollowing background information, “The conformational structure ofproteins is a key to understanding their biological functions and toultimately designing new drug therapies. The conformational structuresof proteins are conventionally determined by x-ray diffraction fromtheir crystals. Unfortunately, growing protein crystals of sufficienthigh quality is very difficult in most cases, and such difficulty is themain limiting factor in the scientific determination and identificationof the structures of protein samples. Prior art methods for growingprotein crystals from super-saturated solutions are tedious andtime-consuming, and less than two percent of the over 100,000 differentproteins have been grown as crystals suitable for x-ray diffractionstudies.”

International Patent No. WO0109595 A2 for a method and system forcreating a crystallization results database to Lansing Stewart et al.,published Feb. 8, 2001, provides the following background information,“Macromolecular x-ray crystallography is an essential aspect of moderndrug discovery and molecular biology. Using x-ray crystallographictechniques, the three-dimensional structures of biologicalmacromolecules, such as proteins, nucleic acids, and their variouscomplexes, can be determined at practically atomic level resolution. Theenormous value of three-dimensional information has led to a growingdemand for innovative products in the area of protein crystallization,which is currently the major rate limiting step in x-ray structuredetermination. One of the first and most important steps of the x-raycrystal structure determination of a target macromolecule is to growlarge, well diffracting crystals with the macromolecule. As techniquesfor collecting and analyzing x-ray diffraction data have become morerapid and automated, crystal growth has become a rate limiting step inthe structure determination process.”

United States Patent Application No. 2003/0150375 for automatedmacromolecular crystallization screening to Brent W. Segelke, BernhardRupp, and Heike, I. Krupka, published Aug. 14, 2003, provides thefollowing state of technology information, a system of automatedmacromolecular crystallization screening of a sample. Initially, reagentcomponents are selected from a set of reagents and a set of amultiplicity of reagent mixes are produced. A multiplicity of analysisplates are produced utilizing the reagent mixes wherein each analysisplate contains a set format of reagent mixes combined with the sample.The analysis plates are incubated to promote growth of crystals in theanalysis plates. Images of the crystals are made. The images areanalyzed with regard to suitability of the crystals for analysis byx-ray crystallography. A design of reagent mixes is produced based uponthe expected suitability of the crystals for analysis by x-raycrystallography. If the crystals are not ideal, a second multiplicity ofmixes of the reagent components is produced utilizing the design. Thesecond multiplicity of reagent mixes are used for automatedmacromolecular crystallization screening the sample. The second round ofautomated macromolecular crystallization screening may produce crystalsthat are suitable for x-ray crystallography. If the second round ofcrystallization screening does not produce crystals suitable for x-raycrystallography a third reagent mix design is created and a third roundof crystallization screening is implemented. If necessary additionalreagent mix designs are created and analyzed.

SUMMARY

Features and advantages of the present invention will become apparentfrom the following description. Applicants are providing thisdescription, which includes drawings and examples of specificembodiments, to give a broad representation of the invention. Variouschanges and modifications within the spirit and scope of the inventionwill become apparent to those skilled in the art from this descriptionand by practice of the invention. The scope of the invention is notintended to be limited to the particular forms disclosed and theinvention covers all modifications, equivalents, and alternativesfalling within the spirit and scope of the invention as defined by theclaims.

The present invention provides a robotic CCD microscope and proceduresto automate crystal recognition. The robotic CCD microscope andprocedures enables more accurate crystal recognition, leading to fewerfalse negative and fewer false positives, and enable detection ofsmaller crystals compared to other methods available today. Accuratecrystal recognition, particularly of small crystals, is a recognizedproblem in structural genomics, protein crystallography, and rationalpharmaceutical design. Protein crystallography is projected to be a $1billion industry in 2005. Accurate automated crystal recognitionpromises to substantially reduce the total man hours required to operatestructural genomics processes and increase the throughput of theseprocesses.

The present invention has applicability in Structural Genomicsindustries (e.g., Syrxx, SGX, Plexxikon, etc.). The invention also hasapplicability in rational drug discovery (e.g., Merk, Chiron, Roche,Johnson and Johnson, Sandos, etc.). The invention can also be extendedto numerous other applications involving automated recognition ofmicroscopic scale objects using light microscopy (e.g., Tissuedissection pathology, tissue typing, colony counting, etc.).

The present invention provides a robotic charge-coupled devicesmicroscope for enhanced crystal recognition. In one embodiment thepresent invention comprises a charge-coupled devices camera, a zoom lensthat has gears to drive zoom and focus, a zoom motor, a focus motor, aplate holder (or plate nest), a motorized xy stage, and a lightingsystem. In one embodiment the lighting system is made up of a cluster ofhigh brightness white lumi-LED's. In one embodiment the lumi-LED's arearranged in circle, at the outer edge, and a small constellation oflights in the middle of the circle. In one embodiment the lumi-LED's inthe middle of the circle are covered by a diffuser. In one embodimentthe light is independently switchable and controlled through software.In one embodiment the circle of lights is off axis with a lens thatenables the camera to take darkfield images, where the subject beingimaged is not in a direct line between the light and the lens.

One embodiment of the present invention provides a roboticcharge-coupled devices microscope for enhanced crystal recognitioncomprising a camera that produces light microscopy images, a lightingsystem that produces light, a digital conversion component of the camerathat converts the light microscopy images into corresponding phase-baseddigital image data using Fourier transform; an edge detection componentthat detects edges from the image data by computing local maxima of aphase congruency-related function associated with each image; asegmentation component that divides the detected edges into discreteline segments; and a geometric analyzer component that evaluates thegeometric relationships that the line segments have with each other toidentify any crystal-like qualities, and determines whether crystals arepresent in each image based on the evaluation.

One embodiment of the present invention provides a method of forenhanced crystal recognition for detecting macromolecular crystals inimages comprising detecting edges in the images by identifying localmaxima of a phase congruency-related function associated with eachimage; segmenting the detected edges into discrete line segments;evaluating the geometric relationships that the line segments have witheach other to identify any crystal-like qualities; and determining thepresence of crystals in each image based on the evaluation.

The invention is susceptible to modifications and alternative forms.Specific embodiments are shown by way of example. It is to be understoodthat the invention is not limited to the particular forms disclosed. Theinvention covers all modifications, equivalents, and alternativesfalling within the spirit and scope of the invention as defined by theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated into and constitute apart of the specification, illustrate specific embodiments of theinvention and, together with the general description of the inventiongiven above, and the detailed description of the specific embodiments,serve to explain the principles of the invention.

FIG. 1 illustrates a CCD microscope having a CCD camera, a zoom lensthat has gears to drive zoom and focus, a zoom motor, a focus motor, aplate holder (or plate nest), a motorized xy stage, and a lightingsystem.

FIG. 2 illustrates a ring light for darkfield imaging.

FIG. 3 is an illustration showing a darkfield image of small crystals.

FIG. 4 is an illustration showing an original brightfiled image taken atlow magnification.

FIG. 5 is an illustration showing a light field image after zooming inon objects identified in a darkfield image.

FIG. 6 is an illustration showing an output from crystal recognitionsoftware.

FIG. 7 is an illustration showing a structural genomics pipeline, fromproteins to crystals and to the molecular structure.

DETAILED DESCRIPTION OF THE INVENTION

Referring to the drawings, to the following detailed description, and toincorporated materials, detailed information about the invention isprovided including the description of specific embodiments. The detaileddescription serves to explain the principles of the invention. Theinvention is susceptible to modifications and alternative forms. Theinvention is not limited to the particular forms disclosed. Theinvention covers all modifications, equivalents, and alternativesfalling within the spirit and scope of the invention as defined by theclaims.

Proteomics is the field of bioscience involving the characterization ofthe proteins encoded by the human genome, and enabled by the genesequence data produced by the human genome project. Since the structureof a protein is key to understanding its function, one field ofproteomics in particular has rapidly emerged concerning high throughputstructure determination or structural genomics. In determining proteinstructure, the proteins are first crystallized, and crystals areanalyzed by x-ray diffraction experiments from which x-ray diffractionpatterns are obtained which in turn lead to three-dimensional picture ofthe atomic arrangement in the crystal. Advances in macromolecularcrystallography techniques, computer speed, and the availability ofhigh-energy synchrotron x-ray sources, make rapid structuredetermination possible given adequate quality protein crystals.

Crystal growth, however, is difficult because proteins are large,irregularly shaped molecules that do not readily come together in arepeating pattern, and the complete set of crystallization conditions istoo large and impractical to screen comprehensively. Thus, previouslyuncrystallized proteins must be screened on a trial and error basisagainst a large array of conditions that have the potential to inducecrystal formation. Automated methods using, for example, robotic liquidhandling devices, robotic CCD-based microscope cameras, or lightmicroscopes equipped with robotic stages and CCD cameras, have beendeveloped and are commercially available to speed up the process ofsetting up and recording the results (automated image capture) of alarge number of crystallization trials. However, a practical problemremains in that each experiment must still be visually inspected todetermine successful crystal formation. In fact, the high throughputenabled by the automation in setup and image-capture has increased thevisual inspection bottleneck, which is typically performed manually byhuman intervention.

One example of an automated crystal detection method developed toaddress the visual inspection bottleneck is disclosed in the article“Intelligent Decision Support for Protein Crystal Growth” (by Jurisicaet al, IBM Systems Journal, Vol. 40, No. 2, 2001). In that article, andas shown in FIGS. 3-7 thereof, images of screening results are analyzedusing a two-dimensional Fourier transform. In particular, FIG. 5Cillustrates the Fourier frequency spectrum used in the analysis, andFIG. 5D illustrates an analysis of the spectrum derivatives and circularaverages to provide features information of the image. From this featureextraction and analysis, the outcome of the experiment is classified as,clear drop, amorphous precipitate, phase separation, microcrystals,crystals, or unknown.

Despite such efforts, difficulties in automating (i.e. without humanintervention) crystal detection remain due to such factors as poor imagequality due to noise and low contrast, differences in crystal shapes,poorly formed crystals, etc. With respect to poor image quality,crystals may have less contrast relative to the background than otherobjects or particles. For example, the difference between the crystaland the background based on 256 gray levels is often 15 levels, whereasthe difference for dirt is usually above 40 levels. Additionally, manydifferent crystal shapes exist due to, for example, the existence ofseveral large classes of crystal shape, the picture is a 2-D projectionof a 3-D object, crystal imperfections with faulty edges, and largevariations in crystal size, e.g., ranging from about 10 μm to greaterthan 300 μm. There are also many things on the picture that are not realcrystals, such as dirt, precipitation, quasi-crystals, small drop due tocondensation, and unidentified effects. Additionally, an automatedcrystal detection process must also achieve a high threshold of accuracyby being able to identify virtually all crystals with a lowfalse-positive rate.

Thus in summary, there is a need for an automated crystal detectionmethod and system for inspecting two-dimensional images and successfullydetecting crystals therefrom. An automated solution for crystaldetection, such as implemented by a software program, would be a greatlabor savor by possessing the capability of processing thousands ofimages a day and provide analysis substantially free from falsepositives.

The problem addressed by the present invention is automated crystalrecognition. The approach taken by most previous efforts is to applyedge detection image analysis to images acquired using light microscopyfollowed by further analysis of varying sophistication. Crystalrecognition is a complicated problem because crystals take on a widerange of shapes and sizes (ranging form sub micron size to millimetersin size) and often appear on a complex background. Well formed crystalshave straight edges but the edges may not appear at high contrast withthe background and the background may interrupt the straight edge.Objects sometimes appear in the foreground to confound crystalrecognition as well. Crystal features also often appear at the limit ofresolution of imaging systems used to acquire crystal pictures, in whichcase it is impossible to distinguish straight edges of a crystal.

Automated crystal recognition is an important problem because inspectingimages for the existence of crystals is a tedious and time consuming jobthat quickly becomes the bottleneck in otherwise highly automatedstructural genomics processes. If it was possible to use automation forsetting up crystallization experiments to full capacity, one would needonly 1 full time employee to setup and track experiments but one wouldneed 8 full time employees to inspect images for evidence of crystals.It is apparent that automated crystal recognition would save atremendous amount of time and money and enable higher throughput for theentire process.

Referring now to the drawings and in particular to FIG. 1, oneembodiment of a robotic charge-coupled devices microscope for enhancedcrystal recognition constructed in accordance with the present inventionis illustrated. The robotic charge-coupled devices microscope forenhanced crystal recognition is designated generally by the referencenumeral 100. The CCD microscope 100 is made of a number of componentsfound in many other CCD microscopes. The CCD microscope 100 includes aCCD camera 101, a zoom lens 102 that has gears to drive zoom and focus,a zoom motor 103, a focus motor 104, a plate holder (or plate nest) 105,a motorized xy stage 106, and a lighting system 107.

Referring now to FIG. 2, the lighting system 107 is described in greaterdetail. The lighting system 107 is made up of a cluster of highbrightness white lumi-LED's 201. The LED's 201 are arranged in circle,at the outer edge, and a small constellation of lights in the middle ofthe circle. The lights in the middle of the circle are covered by adiffuser 202. Each of the lights is independently switchable andcontrolled through software. The circle of light that is off axis withthe lens enables the camera to take darkfield images, where the subjectbeing imaged is not in a direct line between the light and the lens.

Darkfield microscopy, in combination with the other hardware andsoftware features, enables a new technique to crystal recognition.Darkfield microscopy can be used to quickly asses if there are crystalspresent because crystal facets reflect more light than other, unfacettedobjects. Because the light is off axis with the lens, much lessscattered light is collected by the lens and the field is “dark.” Thefaceted objects appear on the darkfield image with very high contrast.This is especially useful when looking for very small crystals (by smallwe mean small relative to the magnification). Crystals that are not atall apparent in the brightfield, because they are smaller than theresolution at low zoom, appear as small bright points in the dark fieldimage and are easily identified as a potential crystal. The potentialcrystal can then be centered under the lens and the lens can be zoomedand refocused on the position of the putative crystal. Then abrightfield image is acquired and crystal recognition is applied.

For crystals to be observed in a darkfield image, the crystal facetshave to be oriented in a particular angle relative to the incident lightand the lens. If the experiments are illuminated from all angles toensure any crystal will reflect light in to the lens, there is much morelight scattered in to the lens, reducing the contrast. The presentinvention addresses that by having a ring of switchable lights. Inpractice, a series of images can be accumulated and put together in to acomposite image. Each image is taken with incident light impinging onthe object field from a narrow angle, thereby maintaining a highcontrast.

One aspect of the present invention includes a method of detectingmacromolecular crystals in light microscopy images comprising: detectingedges in said images by identifying local maxima of a phasecongruency-related function associated with each image; segmenting thedetected edges into discrete line segments; evaluating the geometricrelationships that the line segments have with each other to identifyany crystal-like qualities; and determining the presence of crystals ineach image based on said evaluation.

Another aspect of the present invention includes a computerized systemfor detecting macromolecular crystals from light microscopy imagescomprising: a digital conversion component that converts said lightmicroscopy images into corresponding phase-based digital image datausing the Fourier transform; an edge detection component that detectsedges from the image data by computing local maxima of a phasecongruency-related function associated with each image; a segmentationcomponent that divides the detected edges into discrete line segments;and a geometric analyzer component that evaluates the geometricrelationships that the line segments have with each other to identifyany crystal-like qualities, and determines whether crystals are presentin each image based on said evaluation.

Another aspect of the present invention includes a computerized systemfor detecting macromolecular crystals from light microscopy imagescomprising: means for digitally converting said light microscopy imagesinto corresponding phase-based digital image data using the Fouriertransform; means for detecting edges from the image data by computinglocal maxima of a phase congruency-related function associated with eachimage; means for dividing the detected edges into discrete linesegments; means for evaluating the geometric relationships that the linesegments have with each other to identify any crystal-like qualities;and means for determining the presence of crystals in an image from saidevaluation.

Another aspect of the present invention includes a computer programproduct comprising: a computer useable medium having a computer readablecode embodied therein for causing the detection of macromolecularcrystals in light microscopy images, said computer program producthaving: computer readable program code means for causing a computer todetect edges in said images by identifying local maxima of a phasecongruency-related function associated with each image; computerreadable program code means for causing said computer to segment thedetected edges into discrete line segments; computer readable programcode means for causing said computer to evaluate the geometricrelationships that the line segments have with each other to identifyany crystal-like qualities; and computer readable program code means forcausing said computer to determine the presence of crystals in eachimage based on said evaluation.

Another aspect of the present invention includes an article ofmanufacture comprising: a computer useable medium having a computerreadable code means embodied therein for causing the detection ofmacromolecular crystals in light microscopy images, said computerreadable code means in said article of manufacture comprising: computerreadable program code means for causing a computer to detect edges insaid images by identifying local maxima of a phase congruency-relatedfunction associated with each image; computer readable program codemeans for causing said computer to segment the detected edges intodiscrete line segments; computer readable program code means for causingsaid computer to evaluate the geometric relationships that the linesegments have with each other to identify any crystal-like qualities;and computer readable program code means for causing said computer todetermine the presence of crystals in each image based on saidevaluation.

And another aspect of the present invention includes a program storagedevice readable by a machine, tangibly embodying a program ofinstructions executable by the machine to perform method steps fordetecting macromolecular crystals from light microscopy images, saidmethod steps comprising: detecting edges in said images by identifyinglocal maxima of a phase congruency-related function associated with eachimage; segmenting the detected edges into discrete line segments;evaluating the geometric relationships that the line segments have witheach other to identify any crystal-like qualities; and determining thepresence of crystals in each image based on said evaluation.

Referring now to FIG. 3, an illustration shows a darkfield image ofsmall crystals. Some of the crystals are large enough that crystal shapeis obvious, but there are some crystals that are too small to resolvethat show up in the darkfield image as points of light.

Referring now to FIG. 4, an illustration shows original brightfiledimage taken al low magnification to catch the whole experiment. The 400square indicates the region that was zoomed in on to generate the nextpicture.

Referring now to FIG. 5, an illustration shows a Light field image afterzooming in on objects identified in a darkfield image. Note, thecrystals were not resolved in the previous picture but they are apparentat high magnification.

Referring now to FIG. 6, an illustration shows output from crystalrecognition software. Note that the software accurately identifiedcrystals 500 as small as 4 uM in size.

One embodiment of the present invention is identified by the term“CrysFind.” CrysFind is a robust automated protein crystal recognitionsystem that substitutes or replaces the human observer, thus removingthe final roadblock to creating a fully automated crystallizationlaboratory. Nearly all processes in structural genomics have beeneffectively automated—except the critical step of identifying newlyformed protein crystals. Now, the hope and the promise of structuralgenomics—not only to know all human genes but also to know the fullthree-dimensional structure of every gene product—are closer toattainment. The CrysFind system for automated protein crystalrecognition offers the following features:

Full walk-away automation for the inspection of crystallizationexperiments.

Superior method for crystal detection.

Unique integration of hardware and software.

Integrated, iterative process for recognition of small crystals.

Higher resolution, contrast, and signal to noise.

Minimal false negative and false positive rates.

The CrysFind system is the only viable substitute for a human observer,and it surpasses both in speed and in accuracy what a human could dowith a comparable microscope. Using CrysFind, a highly mechanizedlaboratory could produce tens of thousands of experiments a day with asingle attendant. Its impact is far-reaching and includes the fields ofstructural genomics, structural biology, drug discovery, biotechnology,agriculture, proteomics, infectious disease, and basic biology. Data ofthis scale could enable fields of science not yet conceived.

Completion of the Human Genome Project is seen as an important firststep toward a much deeper and more profound understanding of biology. Weare now poised to take the massive sequence data set and drill down tofind out which genes do what and how they do it. Structural genomics wasidentified shortly after the completion of the human genome project asthe next big discovery-based research effort to follow the human genomeproject. Structural genomics was viewed as the potential successor tothe human genome project because nearly all of the essentialtechnologies for pursing structural biology on a massive scale seemed tobe in hand or on the horizon. The hope and the promise of structuralgenomics as a follow-on to the human genome project was that we wouldnot only know all human genes, but we would also know the fullthree-dimensional structure of every gene product. Having the structuresof every gene product, we could begin to develop atomic-level models ofwhat today seem like impossibly complicated biological processes.

Approximately five years into the structural genomics era, it becameapparent that there were several obstacles to achieving the anticipatedpromise. Major barriers were protein crystallization (forming acrystalline substance from a protein solution) and the absence of afully automated crystallization laboratory. The one key missingtechnology needed for a fully automated crystallization laboratory is arobust method for automated crystal recognition. In particular, a methodfor detecting very small crystals (<10 μm) was desperately needed. TheCrysFind system technology for automated crystal recognition provides akey piece of the puzzle for full automation of crystallography. CrysFindcomplements the suite of technologies that made the vision of structuralgenomics possible and moves us a major step forward toward realizing thepromise of structural genomics. Isolated and purified proteins samplesare put into hundreds and perhaps thousands of crystallization screeningtrials. How to predict crystallization is unknown, so many conditionsmust be tried in the hopes of producing a crystal. Crystallizationevents are rare and can be easily overlooked, but each crystal isprecious. If a crystal is discovered, it is used to elucidate the fullthree-dimensional structure of the protein at a resolution sufficient tosee individual atoms. This last part of a structural genomics pipeline,from proteins to crystals and to the molecular structure, is describedin FIG. 7. FIG. 7 illustrates a structural genomics pipeline 700 fromprotein 701 to crystal 702 to atomic structure 703.

Referring now to FIGS. 1-7, the present invention provides an automatedmethod and system for detecting the presence of macromolecular crystalsfrom light microscopy images, such as those obtained fromcrystallography experiments. Generally, the automated method utilizesthe phase information of the pixels in each image in performing crystaldetection. This provides much lower sensitivity to noise and otherartifacts compared to amplitude or gradient based methods and leads toan edge map with many more features. And in particular, the edge mapprovided by the phase congruency method is used to identify specificgeometric features in an image attributable to, and most likelyindicative of, a crystalline structure, such as, for example, paralleledges, of similar length, facing each other, in relatively closeproximity. Evaluation of detected geometries in this manner makespossible a low rate of false-positive detections and an effectiveresolution to the visual inspection bottleneck discussed in theBackground caused by manual inspections. It is notable that the term“automated” suggests the absence of human intervention, althoughoversight of detection performance may still be present.

Implementation of the present invention is suitably achieved using, forexample, software, computer code, ROM, integrated circuit, etc.(hereinafter “software”) to execute and control the method steps andsystem functions. It is appreciated that the software may be written inany suitable programming language for operation on a suitable operatingsystem or platform, not limited to any particular language or operatingsystem or platform. For example, the software may be written as objectoriented code having implementable subroutines that enable the user tocall them in sequence to solve problems.

Two-dimensional images captured from crystallography experiments arefirst provided as input to begin the process. Edge features, if any, ineach image are detected using phase congruency, and in particular, aphase congruency-based function. The detected edges are then segmentedinto discrete, straight line segments, followed by a geometric analysisand evaluation (hereinafter “evaluation”) of the geometric relationshipsthat the line segments have with each other to identify any crystal-likequalities. In particular, geometric relationships, such as parallellines, facing each other, and similarity in length, as well as relativeproximity to each other, are all crystal-like qualities in that they arecharacteristic of crystalline structure. And finally, a determinationand decision is made based on the preceding geometric evaluation, as towhether crystals are present in a particular image. A decision withrespect to any one image may be reported separately, or together in alist with other images determined to have detected crystals. Thecorresponding ones of the experiments may subsequently be collected forX-ray determination of atomic structure and protein identification.

Generally, the edge detection stage uses the phase of the Fouriertransform of the image to find the images. Taking, for example, a 1-Dstep signal (representing the equivalent of an edge) and its cosinedecomposition, the phases of all cosines are equal to zero only at theedge location, while everywhere else, the phases have different values.Since the phases of real signals are never actually equal, phasecongruency or repartition is a method for measuring how much the phasesof all the cosines are equal, with an edge corresponding to a localmaximum of the phase congruency. It is preferred to compute the standarddeviation of the cosine of the phases since computing the standarddeviation of the phases at each point can give wrong results for thesimple reason that 0=2π. The phase congruency function for a 1-Dcontinuous signal is therefore defined as:

$\begin{matrix}{{{PC}(x)} = {\max\limits_{\theta \in {\lbrack{0,{2\pi}}\rbrack}}\frac{\int{a_{\omega}{\cos\left( {{\omega\; x} + \Phi_{\omega} - \theta} \right)}{\mathbb{d}\omega}}}{\int{a_{\omega}{\mathbb{d}\omega}}}}} & \left( {{Equation}\mspace{20mu} 1} \right)\end{matrix}$where α_(ω) and Φ_(ω) are respectively the amplitude and phase of thesignal. Thus identifying the local maxima of the phase congruencyfunction will also identify and thereby detect the edges. It is notable,however, that while this function provides the correct computationalresults, it is slow to compute directly.

In the alternative, the same result can be obtained using local energy,and Gabor filters which are similar in function to those present in thehuman brain for image perception and vision. The local energy of a 1-Dsignal is defined as:

$\begin{matrix}{{{LE}(x)} = \sqrt{{I^{2}(x)} + {H^{2}(x)}}} & \left( {{Equation}\mspace{14mu} 2} \right)\end{matrix}$where I(x) is the 1-D signal, and H(x) is the Hilbert transform of I(i.e. ninety degree phase shift of I(x) in the frequency domain). Giventhe cosine decomposition of I(x) is:∫α_(ω) cos(ωx+Φ _(ω))dω  (Equation 3)then H(x) has the decomposition:−∫α_(ω) sin(ωx+Φ _(ω))dω  (Equation 4)

I(x) is obtained by convolving the original signal by a filter to removethe DC component. And H(x) is obtained by filtering the previous resultby the Hilbert transform of the first filter. The human visual systemhas neuronal structures similar to a pair of odd and even symmetricfilters in quadrature. If M_(e) is the even filter and M_(o) is the oddfilter, one has:

$\begin{matrix}{{M_{e}*{f(x)}} \approx {\int_{- \infty}^{+ \infty}{a_{\omega}{\cos\left( {{\omega\; x} + \Phi_{\omega}} \right)}\ {\mathbb{d}\omega}}}} & \left( {{Equation}\mspace{20mu} 5} \right) \\{and} & \; \\{{M_{o}*{f(x)}} \approx {- {\int_{- \infty}^{+ \infty}{a_{\omega}{\sin\left( {{\omega\; x} + \Phi_{\omega}} \right)}\ {\mathbb{d}\omega}}}}} & \left( {{Equation}\mspace{20mu} 6} \right)\end{matrix}$

The even symmetric filter is chosen so that it covers as much of thefrequency spectrum as possible, and at the same time removing the D.C.term. Since it is nearly impossible to have such a perfect filter, theapproximate equal sign is used in Equations (5) and (6) above. The oddfilter is the π/2 phase shift even one. In the real implementation, thefilters are band-pass filters and different local energies are computedfor different scale. This allows for a multi-scale analysis of the imageto be performed, with the possibility to look for features of differentsize, and for example getting rid of noise or features very small thatcannot be crystal.

The phase congruency function and the local energy function are relatedby:

$\begin{matrix}\begin{matrix}{{{LE}(x)} = {\left( {{\int_{- \infty}^{+ \infty}{a_{\omega}{\cos\left( {{\omega\; x} + \Phi_{\omega}} \right)}\ {\mathbb{d}\omega}}}, -} \right.}} \\{{\int_{- \infty}^{+ \infty}{a_{\omega}{\sin\left( {{\omega\; x} + \Phi_{\omega}} \right)}\ {\mathbb{d}\omega}}}} \\{= {\max\limits_{\theta \in {\lbrack{0,{2\pi}}\rbrack}}\left( {{\int_{- \infty}^{+ \infty}{a_{\omega}{\cos\left( {{\omega\; x} + \Phi_{\omega}} \right)}\ {\mathbb{d}\omega}}}, -} \right.}} \\{\left. {\int_{- \infty}^{+ \infty}{a_{\omega}{\sin\left( {{\omega\; x} + \Phi_{\omega}} \right)}\ {\mathbb{d}\omega}}} \right) \cdot \left( {{\cos\;\theta},{{- \sin}\;\theta}} \right)} \\{= {\max\limits_{\theta \in {\lbrack{0,{2\pi}}\rbrack}}{\int_{- \infty}^{+ \infty}{{a_{\omega}\begin{pmatrix}{{{\cos\left( {{\omega\; x} + \Phi_{\omega}} \right)}\cos\;\theta} +} \\{{\sin\left( {{\omega\; x} + \ \Phi_{\omega}} \right)}\sin\;\theta}\end{pmatrix}}{\mathbb{d}w}}}}} \\{= {\max\limits_{\theta \in {\lbrack{0,{2\pi}}\rbrack}}{\int_{- \infty}^{+ \infty}{a_{\omega}\left( {{\cos\left( {{\omega\; x} + \Phi_{\omega} - \theta} \right)}\ {\mathbb{d}w}} \right.}}}}\end{matrix} & \left( {{Equation}\mspace{20mu} 7} \right) \\{{{LE}(x)} = {\int{a_{\omega}{{\mathbb{d}\omega} \cdot {{PC}(x)}}}}} & \left( {{Equation}\mspace{20mu} 8} \right)\end{matrix}$

As used herein, the term “phase congruency-based function” is used todescribe both the phase congruency function PC(x) as well as the localenergy function LE(x). A local maximum in the local energy correspondsto a local maximum in the phase congruency, and to an edge. Therefore,in order to search for local maxima in the phase congruency function,one equivalently searches for local maxima in the local energy function.These local maxima will occur at step edges of both parity (up or down),lines and bar edges, and other types of features such as the illusionpatterns mentioned before. While the previous calculations for bothPC(x) and I(x) were for 1-D signals, it is appreciated that a 2-D signalcan also be decomposed in a series of 1-D signals by traversing andaccounting for many orientations. The local energy of a point will bethe maximum local energy among all the orientations.

An exemplary edge detection step using a phase congruency-based function(e.g., local energy function) associated with an image will bedescribed. Multiple orientations are considered. For each orientationand each scale, a Gabor filter is used to filter the image. Theobjective of this filtering step is to convert the spatial informationin another space representation where the noise will be presenteverywhere, and the information will be more localized, so that thenoise and information can be separated, and converted back to the normalspace. Gabor filters are used since they closely approximate the humanvisual system, as previously discussed. They are the combination of thetwo real filters in quadrature, and are the result of a sine wavemodulated by a Gaussian function as in:

$\begin{matrix}{{g\left( {x,y} \right)} = {\frac{1}{2{\pi\sigma}^{2}}\exp\left\{ {- \frac{x^{\prime 2} + y^{\prime 2}}{2\sigma^{2}}} \right\}{\cos\left( {2{\pi\omega}\; x^{\prime}} \right)}}} & \left( {{Equation}\mspace{20mu} 9} \right)\end{matrix}$where (x′, y′)=(xcosθ+ysinθ, −xsinθ+ycosθ). In other words, (x′, y′) isthe θ rotation of (x, y). And local energy is calculated from theinformation at the different scales.

Noise variance and mean of the local energy is estimated, and a noisethreshold is applied on the local energy. Considering first the noisethreshold, the filter response will be decreased by the noise thresholdto remove the components considered as noise. However, finding the rightthreshold can be difficult. The expected response of the Gabor filtersto a pure noise signal must first be examined. If the noise is supposedto be Gaussian, the complex response will have a 2-D Gaussiandistribution. And the magnitude will be a Rayleigh distribution as:

$\begin{matrix}{{R(x)} = {\frac{x}{\sigma_{g}^{2}}{\mathbb{e}}^{\frac{- x^{2}}{2\sigma_{g}^{2}}}}} & \left( {{Equation}\mspace{20mu} 10} \right)\end{matrix}$where σ_(g) ² is the variance of the 2-D Gaussian distribution. The meanand variance of the Rayleigh distribution are given by:

$\begin{matrix}{{\mu_{r} = {\sigma_{g}\sqrt{\frac{\pi}{2}}}}{and}{\sigma_{r}^{2} = {\frac{4 - \pi}{2}\sigma_{g}^{2}}}} & \left( {{Equation}\mspace{20mu} 11} \right)\end{matrix}$

The noise threshold can be set to some number of standard deviationbeyond the mean of the distribution as in T=μ+kσ_(r).

Additionally, the noise amplitude distribution must be determined. Themedian value is considered as a statistically robust estimation of themean. The smallest scale of the Gabor filter is used because the noiseshould be the most present at this scale. The median of a Rayleighdistribution is the value x such that

$\begin{matrix}{{\int_{0}^{x}{\frac{x}{\sigma_{g}^{2}}\ {\mathbb{e}}^{\frac{- x^{2}}{2\sigma_{r}^{2}}}}} = \frac{1}{2}} & \left( {{Equation}\mspace{20mu} 12} \right)\end{matrix}$that leads to

$\begin{matrix}{{median} = {\sigma_{g}\sqrt{{- 2}\;{\ln\left( {1/2} \right)}}}} & \left( {{Equation}\mspace{14mu} 13} \right)\end{matrix}$

The mean of the Rayleigh distribution being σ_(g)√{square root over(π/2))}, one gets the estimated mean of the amplitude response atsmallest scale by a simple proportionality

$\begin{matrix}{{mean} = {\frac{\sigma_{g}\sqrt{\pi/2}}{\sigma_{g}\sqrt{{- 2}{\ln\left( {1/2} \right)}}} \cdot {median}}} & \left( {{Equation}\mspace{20mu} 14} \right)\end{matrix}$

Assuming that noise has a uniform repartition among frequency, the noisethreshold at larger scales can be deduced by the threshold at thesmallest scale since the noise amplitude response will be proportionalto the bandwidth, itself related to the scale. It is notable that whilethe noise reduction process is described as executed in the edgedetection stage, it is appreciated that noise reduction may be performedas a pre-processing step prior to edge detection.

Local energy from each orientation is summed together, and the magnitudefrom each orientation is summed together. The summations provide thevalues necessary to solve Equation (8) above. In particular, by dividinglocal energy by the magnitude, the phase congruency function isobtained. Furthermore, local maxima of the phase-congruency-basedfunction (either PC(x) or LE(x)) is identified to detect correspondingedges. In this manner an edge map may be constructed, for example, withlocal maxima occurring at step edges of both parity (up or down), linesand bar edges, and other types of features such as the illusionpatterns.

Additionally, the geometry analysis and evaluation stage may also beperformed using additional criteria, including: (1) minimum gradientalong edges, and (2) central symmetry. With respect to the firstadditional criteria, and for a given edge, the gradient (i.e. firstderivative of the original picture) for each pixel overlapping this edgeis summed together. If this sum is above a specific gradient, the edgeis then considered relevant. With respect to the second criteria,symmetry centers are found using the phase information. An importantaspect of symmetry is the periodicity that it implies in the structureof the object. To determine the centers of symmetry, the picture isfiltered with the same pair of even and odd symmetric Gabor filters asfor the edge detection technique. At a point of symmetry, the absolutevalue of the even filter will be large, and the absolute value of theodd one will be small. Taking the difference of the absolute value ofthe even-symmetric filter and the odd one give a quantification of thedegree of symmetry in an object.

The Promise of Structural Genomics—Structural genomics holds the promiseto provide a profound new understanding of biology. Structural genomicsaims to produce protein structures on a massive scale—a scale akin tothat of the human genome project. The advent of structural genomicsmarks a new era for structural biology in drug discovery. The success ofthe human genome project and significant advances in new technologiesfor structural biology have made possible a new grander vision for whatis possible using structural biology as a means to interrogate nature'sprocesses.

Structural genomics has already spawned new industries. Companies havesprung up to sell new technologies or consumables that aid structuralgenomics efforts, and companies have also been launched with the missionto use structural genomics as a powerful new tool to speed the time fordeveloping new medicines to reach the marketplace.

The Automated Protein Crystal Recognition System—The CrysFind system,developed for automated crystal recognition, is fairly robust fordetecting larger crystals (<50 μM). CrysFind reduces both the falsenegative rate and false positive rate to <3% compared to a trained humanobserver. However, criterion for a robust crystal detection procedurethat could be fully substituted for a human observer is <5% falsenegatives on crystals smaller than 10 μM since this is better than thethreshold achievable by a trained human observed using current systems.

The CrysFind system imaging system is capable of acquiring images at avery high magnification, which enables detection of small crystals (<5μM), but a whole experiment cannot be interrogated at maximum zoombecause image acquisition and image analysis becomes rate limiting. Whatwas needed was a procedure that quickly identified prospective crystalsat low magnification and confirmed the presence of crystals at highzoom, effectively zeroing in on just the important area of the imagefield. Small potential crystals are rapidly identified at low zoom usingdark-field imaging (FIG. 3). Next, the bright features are centered, andthe size of the image is enhanced by the automated zoom option (FIG. 5).Then, a high-zoom bright-field image is captured. Lastly, CrysFindautodetection is applied (FIG. 6) to detect if crystals are present.

FIG. 3 shows Dark-field image of small crystals. Some of the crystalsare large enough that the crystal shape is obvious, but there are somecrystals that are so small they cannot be resolved. The crystals thatare smaller than the resolution at the given zoom show up in thedark-field image as points of light.

FIGS. 5 and 6 are Photographs showing (a) a high-zoom image of smallcrystals taken with CrysFind's robotic CCD imaging system and (b) theresults of automated crystal recognition with CrysFind. Any line in theoutput indicates that CrysFind has found a crystal.

The combination of switchable lighting for dark-field and bright-fieldimaging, auto-focus, auto-zoom, and software for crystal recognitionfacilitates this procedure for robust automated crystal recognition ofsmall crystals. All of these features are integrated into the CrysFindsystem. This is the only available system that is a viable substitutefor a human observer and, therefore, the only product of its kindhelping to fulfill the grander vision of structural genomics. TheCrysFind system frees valuable human resources, but more importantly,identifies precious leads that are missed with other methods. Each newlead is the potential lynch pin to the development of a new therapy, anew antibacterial, or the basis for a new understanding of biology.

The CrysFind system has a number of specifications or capabilities thatare directly comparable and superior to capabilities offered bycompetitive products, but the CrysFind system delivers combined featuresthat enable a procedure for automated protein crystal recognition thatno other product offers. The CrysFind system is the only viablesubstitute for a human observer, surpassing both in speed and inaccuracy what a human could do with a comparable microscope. TheCrysFind system for automated protein crystal recognition offers thefollowing:

Full walk-away automation for the inspection of crystallizationexperiments.

A complement to structural genomics technologies.

Greater time savings.

Increased screening capacity.

Faster delivery time of new therapies to the consumer.

These benefits are enabled by the following capabilities:

Superior method for crystal detection.

Unique integration of hardware and software.

Integrated iterative process for recognition of small crystals.

CrysFind has several technical specifications that surpass capabilitiesof one or all other systems available as well. The CrysFind systemautomated protein crystal recognition system delivers the followingfeatures:

Higher resolution.

Higher signal to noise.

Higher contrast.

Improved false negative and false positive rates.

Full walk-away automation from image capture through crystal detection.

Structural Genomics—The CrysFind system product will be principallyapplied to structural genomics. For the promise of structural genomicsto be achieved—not only to know all human genes but also to know thefull three-dimensional structure of every gene product—automation mustfacilitate crystallography on a massive scale. The CrysFind systemautomated protein crystal recognition system does exactly that becauseit is the only effective substitute for a human observer. With thisproduct, it is now possible that a highly mechanized laboratory couldproduce and attend to tens of thousands of experiments a day with asingle attendant. As with the human genome project, producing data onthis scale will enable fields of science yet to be conceived.

Structural Biology—The CrysFind system also has great value in a moretraditional structural biology laboratory. While structural genomicssettings will put the highest demands on technologies for accuracy andthroughput, hundreds of more traditional structural biology laboratorieswould also benefit from the CrysFind system. The CrysFind systemsurpasses the capabilities of a human observer in speed and in accuracyand could help individual researchers do a more effective job.

Drug Discovery—Robust, effective, and high-throughput crystalrecognition forms the basis for all applications of structural genomics.Structural genomics in turn brings us in to a new era of drug discoveryby making use of the structural information of proteins for thediagnosis, prevention, and treatment of disease. We are entering aperiod of biology research in which it will be commonplace to have anatomic structure for many, if not most, of the proteins, or molecularmachines, encoded in the genome. Over the last 25 years, atomicstructures have been used more and more extensively as the basis fordiscovering new medicines. HIV protease inhibitors, the most effectivetreatment today for HIV AIDS, were derived from the knowledge of the HIVprotease protein atomic structure. As structural genomics provides aquicker way to obtain new structure information and as the database ofknown structures grows, structure-based drug design will becomeincreasingly useful. We can expect a veritable bonanza of new medicinesin the coming decades.

Biotechnology—There is an immediate application in biotechnology.Several new companies have formed, with various business models, usinghigh-throughput crystallography as their platform. The first and mostprevalent business model for startup using high-throughputcrystallography is a service model whereby a company with the criticalinfrastructure convinces big pharma companies to outsourcecrystallography. High-throughput crystallography companies that becomeestablished pursue strategic alliances where they develop criticalexpertise and knowledge of particular disease pathways and perhaps evendevelop lead drug candidates of interest to pharma companies. There iseven a larger number companies, either newly founded to sell patentedtechnologies or entering high-throughput crystallography as a newventure, selling technologies and consumables to other companies withhigh-throughput crystallography platforms. Finally, there is a majorbusiness opportunity to use the power of structure-aided design to buildnanomachines (structures 100 nanometers in size or smaller) andindustrial catalysts.

Agriculture—There is a major business opportunity to use the power ofstructure-aided design to better understand plant biology and to developnew organic-based pesticides that are not harmful to plants or humans.

Proteomics—assigning function to genes with unknown function—High-throughput crystallography can be applied to gene products,proteins of unknown function in a wholesale manner. This application isonly possible due to the tremendous capacity provided by automatedprocesses.

Infectious disease—High-throughput crystallography can be applied to thestudy of infectious bacteria, which will lead to specific antibacterialsor general antibiotics.

Basic Biology—A long-standing recognized problem in bioscience is theso-called “protein folding problem.” The challenge is to understand howa gene sequence encodes a particular protein structure and ultimatelythe function of the protein. Many hundreds of millions of dollars havebeen spent trying to approach this problem theoretically and bycomputation over the last two decades. In fact, the budgets for buildingsome of the world's largest computers are justified, to some degree atleast, on the basis that these computers will address the proteinfolding problem.

With a platform for finally achieving high-throughput crystallography,the protein folding problem may be solved experimentally. If so,biologists will worry less about obtaining protein structure informationand more about understanding biology and will develop more sophisticatedmodels of structure and function at the atomic level.

While the invention may be susceptible to various modifications andalternative forms, specific embodiments have been shown by way ofexample in the drawings and have been described in detail herein.However, it should be understood that the invention is not intended tobe limited to the particular forms disclosed. Rather, the invention isto cover all modifications, equivalents, and alternatives falling withinthe spirit and scope of the invention as defined by the followingappended claims.

1. A robotic charge-coupled devices microscope for enhanced crystalrecognition, comprising: a charge-coupled devices camera, a zoom lensthat provides zoom and focus for said charge-coupled devices camera,said zoom lens having a lens axis, a plate holder, a motorized stage,and a lighting system having an off axis portion that is off axis withsaid lens axis; wherein said lighting system is made up of a cluster ofhigh brightness lights arranged in circle, a small constellation oflights, and a diffuser; wherein said off axis portion of said lightingsystem comprises said cluster of high brightness lights arranged in acircle, wherein said small constellation of lights are in the middle ofsaid circle, and wherein said small constellation of lights in themiddle of said circle are covered by said diffuser.
 2. The roboticcharge-coupled devices microscope of claim 1 wherein said cluster ofhigh brightness lights is made up of a cluster of high brightness whitelumi-LED's.
 3. The robotic charge-coupled devices microscope of claim 2wherein said high brightness white lumi-LED's are arranged at the outeredge of said circle and said small constellation of lights are in themiddle of said circle.
 4. The robotic charge-coupled devices microscopeof claim 3 wherein said high brightness white lumi-LED's areindependently switchable.
 5. The robotic charge-coupled devicesmicroscope of claim 3 wherein said high brightness white lumi-LED'scomprise individual lights and each of said individual lights isindependently switchable and controlled through software.
 6. The roboticcharge-coupled devices microscope of claim 1 wherein said roboticcharge-coupled devices microscope is adapted to image a subject andwherein said cluster of high brightness lights arranged in a circle isoff axis with said lens axis and wherein said off axis portion of saidlighting system enables said camera to take darkfield images, where thesubject being imaged is not in a direct line between said cluster ofhigh brightness lights and the lens.
 7. The robotic charge-coupleddevices microscope of claim 1 including darkfield microscopy, incombination with the other hardware and software features, enables a newtechnique to crystal recognition.
 8. The robotic charge-coupled devicesmicroscope of claim 1 wherein said zoom lens includes gears to drivezoom.
 9. The robotic charge-coupled devices microscope of claim 1wherein said zoom lens includes gears to drive focus.
 10. The roboticcharge-coupled devices microscope of claim 1 wherein said zoom lensincludes gears to drive zoom and focus.
 11. The robotic charge-coupleddevices microscope of claim 1 wherein said zoom lens includes a zoommotor.
 12. The robotic charge-coupled devices microscope of claim 1wherein said zoom lens includes a focus motor.
 13. The roboticcharge-coupled devices microscope of claim 1 wherein said zoom lensincludes a zoom motor and a focus motor.
 14. The robotic charge-coupleddevices microscope of claim 1 wherein said plate holder is a plate nest.15. A robotic charge-coupled devices microscope for enhanced crystalrecognition, comprising: a camera that produces light microscopy images,said camera having a lens with a lens axis, a lighting system thatproduces light, said lighting system having an off axis portion that isoff axis with said lens axis; wherein said lighting system is made up ofa cluster of high brightness lights arranged in circle, a smallconstellation of lights, and a diffuser; wherein said off axis portionof said lighting system comprises said cluster of high brightness lightsarranged in a circle, wherein said small constellation of lights are inthe middle of said circle, and wherein said small constellation oflights in the middle of said circle are covered by said diffuser, adigital conversion component of said camera that converts said lightmicroscopy images into corresponding phase-based digital image datausing Fourier transform; an edge detection component that detects edgesfrom the image data by computing local maxima of a phasecongruency-related function associated with each image; a segmentationcomponent that divides the detected edges into discrete line segments;and a geometric analyzer component that evaluates the geometricrelationships that the line segments have with each other to identifyany crystal-like qualities, and determines whether crystals are presentin each image based on said evaluation.
 16. A method of for enhancedcrystal recognition for detecting macromolecular crystals in imagescomprising the steps of: darkfield imaging the macromolecular crystalsto produce a darkfield image having features, zooming on said featuresin said darkfield image to produce a zoomed image, bright field imagingsaid zoomed image, detecting edges in said zoomed image by identifyinglocal maxima of a phase congruency-related function associated with saidzoomed image; segmenting said detected edges into discrete linesegments; evaluating the geometric relationships that said line segmentshave with each other to identify any crystal-like qualities; anddetermining the presence of crystals in said zoomed image based on saidevaluation.
 17. A method of for enhanced crystal recognition fordetecting macromolecular crystals, comprising the steps of: darkfieldimaging the macromolecular crystals to produce a darkfield image withfeatures, zooming on said features in said darkfield image to produce azoomed image, bright field imaging said zoomed image, and providingcrystal recognition in said zoomed image for detecting macromolecularcrystals using a computer program.
 18. The method of for enhancedcrystal recognition of claim 17 wherein said step of providing crystalrecognition in said zoomed image for detecting macromolecular crystalsusing a computer program uses the CrysFind computer program.
 19. Themethod of for enhanced crystal recognition of claim 17 wherein said stepof providing crystal recognition for detecting macromolecular crystalswith a computer program uses a robust automated protein crystalrecognition computer program.
 20. The method of for enhanced crystalrecognition of claim 17 wherein said step of providing crystalrecognition in said zoomed image for detecting macromolecular crystalsincludes the steps of detecting edges in said zoomed image byidentifying local maxima of a phase congruency-related functionassociated with said zoomed image; segmenting said detected edges intodiscrete line segments; evaluating the geometric relationships that saidline segments have with each other to identify any crystal-likequalities; and determining the presence of crystals in said zoomed imagebased on said evaluation.